by vunderba on 1/22/2024, 4:11:32 AM
All the major smart speaker manufacturers have plans to eventually back their services using LLM's.
https://www.theverge.com/2023/9/20/23880764/amazon-ai-alexa-...
I've already put one together for personal usage using the ESP32-S3. The advantage of using something like this instead of a raw raspberry PI is that you've got a basic far field microphone, screen, wake word support, etc. and then it's just a matter of wiring up the voice to go to deep whisper for recognition, pass it on to a large language model (in my case I'm using mistral.), generate TTS using Mycroft, and sending it back.
Biggest annoyance is having to have a dedicated server, since the ESP is simply not powerful enough to do real time voice recognition and LLM inference.
It's relatively easy to do - I had a workable prototype within a weekend.
by freddealmeida on 1/22/2024, 5:38:41 AM
A number of companies are and have tried to build this. I myself did once in 2018. Alyc.ai if you care to look. We had voice, vision, llm, domain specific training (called fine tuning today I suppose), emotion detection, gesture detection, pose estimation, action recognition (ie. guy is drinking a beer), multiple stereoscopic cameras, microphone mesh, infrared camera (used for depth and night vision), and used nvidia chipsets (jetson) to run models at the edge. Other models ran in the cloud. Our LLM was about 2B parameters trained on about 800MB of Japanese data. Also our interface was a sophisticated peppers ghost (Today I would use lightfield displays) but it was fun. We didn't have RLHF feedback loops so Alyc was a bit complex. Still people loved her. Covid killed this project.
Creating a device similar to Amazon Echo using a Raspberry Pi and ChatGPT would involve both hardware and software components. Here's a conceptual design:
Hardware Components: Raspberry Pi (4 Model B recommended): Serves as the core computing unit. Microphone: For capturing voice commands. A USB microphone or a module compatible with Raspberry Pi can be used. Speakers: To output the ChatGPT responses. Any standard speaker with a 3.5mm jack or Bluetooth connectivity would work. LEDs/Display (Optional): To indicate the device status or display text responses. Power Supply: To power the Raspberry Pi and peripherals. Software Components: Operating System: Raspberry Pi OS or any compatible Linux distribution. Voice Recognition Software: To convert voice commands to text. Solutions like Google Speech-to-Text, Mozilla DeepSpeech, or similar can be used. ChatGPT Integration: A custom program to send the text input to ChatGPT and receive responses. This would require an API integration with OpenAI's ChatGPT. Text-to-Speech (TTS) Engine: To convert ChatGPT's text responses into speech. Google Text-to-Speech or other TTS systems can be used. Workflow: The microphone captures the voice input. Voice recognition software converts the voice input to text. The text is sent to ChatGPT through a custom program. ChatGPT processes the input and sends back a response. The TTS engine converts the response to speech. The response is output through the speakers. Additional Features: Implement wake word detection to activate the device. Add WiFi/Bluetooth modules for internet connectivity and wireless speaker support. Consider adding a small screen or LED lights for visual feedback. Assembly: Connect the microphone and speakers to the Raspberry Pi. Install and configure all the necessary software components. Enclose the setup in a casing with proper ventilation for the Raspberry Pi. Programming and Configuration: Write a script to tie together voice recognition, ChatGPT API calls, and TTS output. Configure network settings for stable internet access. Test and optimize for better performance and reliability. This setup would essentially create a smart assistant device using Raspberry Pi and ChatGPT, offering functionalities similar to Amazon Echo but with the advanced conversational capabilities of ChatGPT.