Project
| # | Title | Team Members | TA | Documents | Sponsor |
|---|---|---|---|---|---|
| 2 | REALTIME PLUG AND PLAY INPUT DEVICE BASED ON SEQUENCES OF MULTIKEY INPUTS (EMBEDDED STENOGRAPHY) Best Commercial Project |
Haoqing Zhu Rishi Krishnan Soham Karanjikar |
William Zhang | design_document1.pdf final_paper1.pdf proposal1.pdf |
|
| Team members: Haoqing Zhu (haoqing3) Soham Karanjikar (sohammk2) Rishi Krishnan (rishik3) # Problem - The verbally disabled don’t have a good way to communicate with other people (especially those who can’t read sign language) at real time speeds. Average typing speeds range from 50 to 70 wpm and maxes out around 140 wpm, while daily English conversations are around 180 wpm, sometimes going up to 250 wpm. - The common keyboard layout is not suitable for typing long texts efficiently. Even though there are a lot of alternative keyboard layouts that try to maximize typing comfort and efficiency, we still need to worry about spellings, and sequencing the letters correctly at high speeds is a problem for a lot of people. In addition, most parts of the keyboard are unutilized when typing. # Solution - The solution is to use stenography, which generally refers to pressing multiple keys simultaneously on a special keyboard. The keys on a steno keyboard are a lot closer to the home position, causing less finger fatigue. Each stroke typically represents one or more syllables, so even long words typically take no more than 3 or 4 strokes. The ability to input at the same or higher speed with lower stroking frequency and less mental burden on spelling makes typing much more efficient. - Traditional steno machines are very expensive, due to their limited use, primarily in court reporting. Open source solutions have been developed recently, but still require software to be installed on the host. This project aims to move that process onto a keyboard, making steno more portable and much easier to use. # Components ## Translation engine The translation engine needs to convert strokes to keyboard output according to user defined entries. The entry can contain plain text, text transformation commands (think auto capitalization after periods, or manual cancelation of spaces before the next entry), arbitrary key codes, and Unicode characters. The translation should match the longest sequence of strokes. Most of this has been implemented, except some parts of the command set, and Unicode and lower/keep cases are not properly handled, and the engine would sometimes crash. ## Input source The input source needs to be able to handle multiple key press/releases at the same time and translate them to the corresponding steno keys. This includes both the physical device and the matrix scanning. A prototype device has been designed and assembled, and matrix scanning and USB functionalities are handled by [QMK](https://qmk.fm) (a open source keyboard firmware). ## Dictionary storage driver The driver needs to be able to provide random access into the dictionary at soft realtime speeds and edit the dictionary. This also includes both the physical device and the firmware module. Currently, the storage device can either be a QSPI flash or a SD card. The dictionary is stored in the drive as a hash tree, and performant reading from the storage is supported. ## User interface This includes interface that let the user edit the dictionary, check the status of the keyboard (for example battery level if using a BLE board), etc. Simple OLED display of last stroke and entry, and battery level have been implemented. ## Dictionary compiling/bootstrapping utility A utility that bootstraps/compiles a dictionary in Plover JSON format into the specified on board dictionary format. This is currently done, but may need to be modified for future format changes. # Success criterion As a prototype has already been made, the success criterion would be an extent to which we improve the system. Currently the planned features/improvements are: - Properly handling Unicode encoding and input in text entries and lower/keep cases. - Implementing in place dictionary editing, including corresponding user interface, in the current dictionary format. - Implementing dictionary loading using Mass Storage Class for AVR version - Implement the full set of commands/meta items present in the Plover dictionary format - Use larger screen for better user interface If we have more time than planned, these are the advanced features we are going to implement: - Dictionary loading using MSC for nRF - Design and implement more general purposed (suitable for reading and editing, maybe searching) dictionary format, and reimplement dictionary bootstrapping/compiling utility for new format # Contingency Plans When/If we need to move online, the challenge we would face would almost be the same as what we have now, which is the difficulty to get hardware for testing. This is something we can overcome as we can meet each other on campus even if not permitted to meet in the ECEB lab. In addition, at least one of us has the lab equipment needed (soldering iron, oscilloscope, etc.) So we are not too worried about transitioning to fully online as we are having the same struggles even with the current situation where classes are in-person. |
|||||