ECE/CS 498: Smartphone Computing and Applications (Fall 2016)
From linear algebra and probability ... to sensing techniques ... to applications, such as localization, gesture recognition, augmented reality, drones, etc.

mobile sensing

Course Description: This course will teach a variety of techniques and algorithms crucial to understanding and developing mobile systems and applications. Topics of interest include:
  1. GPS and indoor location: Why don't we have indoor navigation today ... what are the challenges?
  2. Activity and gesture recognition: Can wearable devices (like smartwatches) track our movements and behavior?
  3. Augmented reality: How can sensing augment computer vision for AR/VR applications ...
  4. Drones, smart toys, and sports analytics: Opportunities when everything in our lives have sensors inside them ...
  5. Privacy: What information is leaking? Can one sensor behave like another? Can you see using sound or WiFi?
  6. New modes of communication: What are the emerging needs for mobile communication? Vibrations? Light? Sound?
  7. Topics of popular choice: Students decide on a new topic or revisit one of the above topics in more depth ...
The techniques/algorithms underpinning these applications will not be taught in isolation, but will be closely motivated by real-world data and use-cases ... so that students always recognize why the mathematics is necessary. The learnt techniques will then be plugged back to understand how a complete system should come together on smartphones, wearables, quadcopters, or other mobile platforms. Overall, this course is about learning to build systems systematically -- through a mix of programming, creativity, and basic mathematics. See more about the course organization below.

Who should take this course?
  • The course is particularly designed for CS/ECE students who find themselves strong in programming but feel rusty (or somewhat distant) from mathematical techniques (possibly because they have not used/needed them for a long time). For these students, this course will serve as a valuable refresher and also help them design systems more systematically.
  • Students with strong theoretical background may not find this course challenging, however, if system design and implementation are of interest, the course can be useful.

Time and Location:   M/W 3 to 4:20pm @ 2015 ECE Building
Instructor:                  Romit Roy Choudhury (
Office hours:              M/W 4:20 to 5:00pm

Course TA:               Mahanth Gowda <>
                                 Ashutosh Dhekne <>

Some more details:
  • Prerequisites: Probability, linear algebra, programming maturity on any one language needed.
  • Experience on mobile platforms a plus but not mandatory: Android tutorials will be offered.
  • No text book required -- materials will be in the form of slides and handouts.
  • Coarse load entails (1) Mini programming assignments (MATLAB, Andoid, iOS, C); (2) reading assignments, and writing reviews; (3) one major project
  • Mobile devices for projects (e.g., smartphones, Google glass, wearables, drones) can be provided.

Grading Information:
Homework and reviews:         10%
Programming assignment:       25%
1 mid-term exam:                    25%
Final Project:                           40%

Project Teams:

1. Indoor Localization Using BLE Beacon Landmarks -- proposal
Pujan Dave, Zack Rhodes

2. Daily Activity Recognition from Smartwatches -- proposal
Nitin Verma, Suraj Sinha

3. Smartphone mouse -- proposal
Daniel Bang, Yunyoung Roh

4. Phone Location on the body: Human body model and decision tree -- proposal
Kaixuan Lin, Fangxiaozhi Yu

5. Bicycle Recognition -- proposal
Khalique Ahmed, Doug MacNerland

6. BuildingSense -- proposal
Ziheng Zeng, Arjun Nijhawan

7. Mobile Device Control Through Knock Recognition & Localization -- proposal
Todd Nguyen, Alvin Wu

8. Indoor localization using 2D-dead reckoning constrained PCA movement model -- proposal
Guangzhe Cui, Yuanyuan Dong

9. Room Cube -- proposal
Aaron Mann, Yifang Zhang

10. Augmented reality using 3D Dead Reckoning -- proposal
Namit Girdhar, Wilfredo Salvador

Course Calendar (subject to change)



intro.ppt ]
Lecture 0.1: Overview of the course

Lecture 0.2: Introduction and motivation ... best practices during the course ... thoughts on projects, etc.

0.1 Submit: none

0.2 Submit: none

Wireless Radio

[ wireless.ppt ]

[ rate_antenna.ppt ]
Lecture 1.1: Wireless basics and WiFi case study (MACAW).
(Wireless channel, CSMA/CA, TDMA, hidden and exposed terminals, SINR, carrier sensing, backoff ...)

Lecture 1.2: WiFi rate selection (RBAR) ... Overview of Bluetooth (BLE), RFID, and IoT beacons
(SNR, BER, symbols, pathloss, multipath, fairness,  ...)

Optional readings: Directional antennas (DMAC), WiFi Energy (SleepWell), Bluetooth low energy (BLE)

1.1 Submit: none

1.2 Submit:
reviews for MACAW and RBAR

Sep 1, 6:30pm in class
Android Tutorial by TA (Ashutosh Dhekne)

Mini project 1: Released. [pdf]
-- Sensor data collection from smartphones
-- Step count estimation and basic activity recognition
Released Mon Aug 29
Due (Extended final): Fri Sep 16, 11:59pm
Some References:
Manifest file and giving permissions to your application: here
Capturing sensor data: here
Saving data to a file: here
We used Android Studio, Java JDK 1.8. Set your JAVA_HOME and JDK_HOME environment variables to point to the JDK folder.

Outdoor GPS Navigation

gps.ppt ]
Lecture 2.1: Basics of GPS positioning
(Linear Systems, solving Ax=b, vector spaces, rank, independence, basis, dimension, least squares, trilateration, triangulation, TDoA)

Lecture 2.2: Differential GPS and emerging problems in drones (SafetyNet)
(Carrier phase, integer ambiguity, rotation matrices, glimpse of randomness, glimpse of filtering ...)

2.1 Submit: none ... but read the Linear_Algebra_primer

2.2 Submit: review for SafetyNet
(skip section 4 and 5)

Mini project 2: Released [pdf]
-- Walking direction estimation
-- Trajectory tracking
-- Communication with physical vibrations
Released: Fri, Sep 23, 2016
Due: Sun Oct 9, 2016, 11:59pm

Indoor Localization

[ radar.ppt ]

[ surroundSense.ppt ]

[ unloc.ppt ]

Lecture 3.1: WiFi Signal Strength based Indoor Localization (RADAR)
(Visualize high dimension data, KNN classification,
fading models, radio propagation model)

Lecture 3.2: Sensing the Ambiance for Localization (SurroundSense and MaLoc)
(Random variables, distributions, expectation, conditional probability, Bayes' rule, introduction to graphical models, Kalman & particle filters)

Lecture 3.3: Localization without Relying on any Infrastructure (UnLoc and Zee)
(Law of large numbers, Clustering, Kalman & particle filtering)

Lecture 3.4: Review techniques and brainstorm project ideas (UWB, indoor GPS, sound localization, CSI based, visual, etc.)

Optional readings: Indoor localization from GPS (CoinGPS), localization using lights (LuxaPose) ...

3.1 Submit: review for the RADAR

3.2 Submit:
none ... but read MaLoc

3.3 Submit: reviews for UnLoc and Zee

3.4 Submit: none

Mini project 3:
-- Topic TBD based on class interest

Walking Direction Estimation
[ walk-compass.pptx ]
Lecture 3.5: I am a Smartphone and I can Tell my Users Walking Direction (WalkCompass)

3.5 Submit: none
Communication with vibrations
[ ripple.pptx ]
Lecture 3.6: Communicating through Physical Vibration (Ripple-II, Ripple-I)

3.6 Submit: none
Location Privacy
[ privacy.ppt ]
Lecture 3.7: Location privacy of motion traces (CacheCloak)
(K-anonymity, space-time intersections, Entropy)

3.7 Submit: review for CacheCloak
Activity and Gesture Recognition

[ armTrack.pptx]

[ MoLe.pptx]
Lecture 4.1: Tracking arm motion with smartphones (uWave and PhonePen)
(Time series, Time Warping, DTW)

Lecture 4.2: Tracking arm motion with smart watches (ArmTrack)
(3D orientation, introduction to Hidden Markov Models, Viterbi decoding)

Lecture 4.3: Decoding finger motion from smart watches and Dictionary (MoLe)
(Random variables, joint distributions, marginalization, priors, maximum likelihood estimation)

4.1 Submit: none

4.2 Submit: review for ArmTrack

4.3 Submit: review for MoLe

Final project:
-- Proposal submission deadline

[ witrack.ppt ]

[ swordFight.ppt ]
Lecture 4.4: Gesture recognition through WiFi signals (WiTrack)
(Doppler shift, DFT, FFT, classification)

Lecture 4.5: Phone to phone 3D ranging, walk estimation, and brainstorming (SwordFight)
(PCA, Correlation, time of flight)

4.4 Submit: none

4.5 Submit: review for WiTrack

[ midterm.ppt ]
Midterm (around Nov 2nd week ... date to be finalized later)

Smart Objects and IoT

[ warping.ppt]

[ Buzz.ppt ]

[ ripple.ppt ]

Lecture 5.1: Smart toys and vehicular sensing (Buzz)
(DTW, Decision Trees)

Lecture 5.2: Vibration based communication and sensing (Ripple and VibraPhone)
(Modulation, demodulation, QAM, synchronization, inter-symbol, PN sequence, phase lock, MIMO, RPCA, harmonics, filter design)

5.1 Submit none ... but read Decision_Trees

5.2 Submit: Review for Ripple

Camera Sensor Fusion for Augmented Reality

[ Overlay.ppt ]

[ insight.ppt ]
Lecture 6.1: Augmented reality through sensor fusion (OverLay)
(Image as vectors, features, SIFT, SURF, image compression, Eigen vectors, PCA, Linear programming)

Lecture 6.2: Continuous Vision and InSight ... and other new ideas (InSight)
(Image sensors, energy proportionality, spatiograms, Kalman filter, string matching)

6.1 Submit: review for OverLay

6.2 Submit none

Assorted/Student choice

[ sports.ppt ]

[ robotic.ppt ]

[ sleepwell.ppt ]
Lecture 7.1: Sport analytics and project updates
(AoA, Gyroscope+magnetometer fusion, clock drift, Kalman filter)

Lecture 7.2: Robotic wireless networks
(WiFi, multipath, fading, shadowing, optimal stopping theory)

Lecture 7.3: SleepWell
(CAM, PSM, distributed fair sharing, clock tricks)

7.1 Submit none

7.2 Submit none

7.3 Submit none

Final project demo and presentation (8-11am, Friday, Dec 9)

Course Organization:
The course will begin with various future-facing topics in mobile sensing, sampled from those listed above. Once we have the bigger picture, we will select one application at a time, and zoom into each of the technical pieces of the puzzle. We will discuss various first-cut solutions to each problem and highlight its limitations ... with the goal of fully appreciating the difficulty of the problem. Then, we will dive into learning various mathematical techniques, starting from the first principles of linear algebra and probability. For instance, while discussing "GPS and indoor localization", we will cover techniques such as:
  • [From scratch] Matrices, vector spaces, norms, rank of matrices, over-determined systems, least squares ...
  • [From scratch] Random variables, expectation, Bayes' rule, probability distributions, ...
  • Time of flight estimation under clock uncertaties
  • Maximum likelihood estimation
  • Clustering and classification
  • Introduction to Kalman filters
While discussing "activity and gesture recognition", we will cover:
  • [From scratch] Dynamic programming, graph traversal, Bayesian statistics, Fourier analysis (FFT), ...
  • Data sampling techniques
  • Signal correlation and (dynamic) time warping
  • Principle component analysis
  • Introduction to Hidden Markov Models (HMMs) and particle filters
As we repeat this process for each topic, we will implement some of these techniques in class assignments and attempt to make them work in real life. We will identify where they break down, understand why, and propose modifications. Put differently, we will appreciate the gap between theory and practice and discuss how to tackle these gaps for real systems.

In the second half of semester, when you have picked-up a number of techniques and are starting to use them effectively, you would start conceiving your own application as a final project for the course. You would carefully motivate the project, argue that it is technically non-trivial, and apply some of the learnt techniques to achieve your goal. The semester will end with each group demonstrating their project to the entire class.