Project

# Title Team Members TA Documents Sponsor
69 A Comprehensive Approach to Tumor Detection using RGB, NIR, and Immersive 3D Visualization
Amy He
TJ Shapiro
Zach Mizrachi
Jason Zhang design_document1.pdf
final_paper1.pdf
other1.pdf
ECE 445 Senior Design RFA

A Comprehensive Approach to Tumor Detection using RGB, NIR, and Immersive 3D Visualization

Team Members:
- Zach Mizrachi (zdm3)
- TJ Shapiro (tylers5)
- Yue (Amy) He (yuehe4)

# Problem

The most widely used approach for tumor removal today is traditional surgery, which introduces a host of problems. This traditional method relies solely on the surgeon's visual and tactile feedback, which is subject to human error. The surgeon is also operating on his or her own view of the tumor, which is often limited when the tumor is not easily visible. All of the above can lead to excess damage being done to the patient in order to increase tumor visibility, or accidental damage caused by human error.

# Solution

We propose a camera system meant to assist a surgeon in their removal of a tumor. The system is intended to perform two main tasks: detect the tumor by segmenting it from the surrounding biological material, and reconstruct the detected tumor in 3D. The camera system is small and highly mobile, such as to allow the surgeon to view all areas of the tumor. The presented solution will improve the visual capabilities of the surgeon, allowing for continuous visualization and informed decision making.

The setup: putting some fluorescent drug over the area of interest, and the tissue would reflect NIR light while the tumor wouldn’t. We use that to distinguish between the tumor area and the healthy area via the tumor-detecting pen system. This method has been validated in the pilot study.

Specifically, we intend to visualize the operating surface in real time in the Apple Vision Pro, highlighting the tumor in augmented reality from the NIR. This will allow the surgeon to record the area of interest guided by the highlighting from NIR, contributing to more accurate photos for the tumor reconstruction. Then, in post processing, we will generate a 3D model of the tumor that will allow the surgeon to have a more detailed view of the region of surgical interest.

## Subsystem

### Casing Module
- **Part Name:** 3D Print
- **Part #:** N/A
- **Protocol:** N/A
- **Purpose of Part:** Hold all components rigidly together

### Imaging Module

- **Part Name:** Beam Splitter
**Part #:** Edmund Optics, Family ID #2185, Visible and NIR Plate Beamsplitters
**Protocol:** N/A
**Purpose of Part:** Take in Visible Light, split the beam into two equal beams

- **Part Name:** NIR Filter
**Part #:** 49950 - RT – Raman 785nm Laser Longpass Set
**Protocol:** N/A
**Purpose of Part:** Filter beam for NIR light

- **Part Name:** NIR Sensor
**Part #:** LI-OV5640-MIPI-AF-NIR
**Protocol:** MIPI
**Purpose of Part:** Record NIR signal

- **Part Name:** RGB Filter
**Part #:** Chroma 27040 - Lum
**Protocol:** N/A
**Purpose of Part:** Filter beam for RGB light

- **Part Name:** RGB Sensor
**Part #:** Digikey, 2289-LI-IMX185-MIPI-M12-ND
**Protocol:** MIPI
**Purpose of Part:** Record RGB signal

- **Part Name:** Lens
**Part #:** Edmund Optics, Family ID #1748, Uncoated Double-Convex (DCX) Lenses
**Protocol:** N/A
**Purpose of Part:** Focus the light on the camera sensors

### Processing Module
- **Part Name:** NVIDIA Jetson
- **Part #:** Digikey, 1597-102110417-ND
- **Protocol:** MIPI
- **Purpose of Part:** Image analysis and sensor fusion
- **Additional Notes:**
- Uses SPI to connect the device and the display
- GPU is responsible for generating a 3D representation based on the input data
- Storing the frames from real-time for later processing

### PCB Components
- **Part Name:** IMU
- **Part #:** Digikey LSM6DSO iNEMO™
- **Protocol:** SPI/I2C, and MIPI I3CSM serial interface
- **Purpose of Part:** Record pose information for the camera via 'Structure from Motion' Algorithm. See Software Overview.

### Modeling Module
- **Part Name:** Apple Vision pro
- **Part #:**N/A
- **Protocol:** N/A
- **Purpose of Part:** Communicates with Mac, which is communicating with Jetson. Project 3D reconstruction of tumor detection/biological information via head-mounted display through augmented reality. This will be done using Apple’s proprietary VisionPro platform as well as SwiftUI and ARKit frameworks.



## Hardware Components

Explain what the subsystem does. Explicitly list what sensors/components you will use in this subsystem. Include part numbers.

We look to replicate a similar hardware setup to the parent study of this project. In this, we will work closely with Professor Gruev to ensure a feasible approach to the hardware system.

Software Overview:

For 3D models to be useful in a surgical scenario, we need the reconstruction to have high levels of detail. For this, we prioritize detail over real-time analysis, and look to implement an open source Structure from Motion algorithm. To further improve upon existing algorithms, we intend to fuse IMU data to eliminate the need to estimate camera pose. We believe this will improve the accuracy of our 3D models.

Existing work has shown that implementing IMU within SFM is not only feasible but improves the robustness of 3D models for small objects. In this work, we look to follow a similar approach to existing literature.



# Criterion For Success

Describe high-level goals that your project needs to achieve to be effective. These goals need to be clearly testable and not subjective.

This project can be separated into goals for 3 stages.


Hardware:
3D Print a casing, allowing for adjustment of beam splitter distance to image sensors
assemble all electrical components correctly
Successfully integrate IMU with PCB
Software
Receive and validate all data on NVIDIA Jetson
RGB Data
NIR Data
IMU Data
Filter RGB images by using NIR region of interest
Set up and run open SFM software on NVIDIA
Improve SFM model with IMU
Perform optimal frame selection using IMU
Augmented/Virtual Reality
Establish communication between Jetson and Vision Pro
Set up pass through mode on Vision Pro, with NIR tumor highlighting
View 3D SFM Point Cloud on Vision Pro
Interact with Point Cloud on Vision Pro

Each bullet here is a goal that we would like to achieve over the course of the semester. Given the difficulty of the task, we plan on utilizing the IMU to improve SFM as a final step, after the pipeline to the Vision Pro has been completed.









El Durazno Wind Turbine Project

Alexander Hardiek, Saanil Joshi, Ganpath Karl

El Durazno Wind Turbine Project

Featured Project

Partners: Alexander Hardiek (ahardi6), Saanil Joshi (stjoshi2), and Ganpath Karl (gkarl2)

Project Description: We have decided to innovate a low cost wind turbine to help the villagers of El Durazno in Guatemala access water from mountains, based on the pitch of Prof. Ann Witmer.

Problem: There is currently no water distribution system in place for the villagers to gain access to water. They have to travel my foot over larger distances on mountainous terrain to fetch water. For this reason, it would be better if water could be pumped to a containment tank closer to the village and hopefully distributed with the help of a gravity flow system.

There is an electrical grid system present, however, it is too expensive for the villagers to use. Therefore, we need a cheap renewable energy solution to the problem. Solar energy is not possible as the mountain does not receive enough solar energy to power a motor. Wind energy is a good alternative as the wind speeds and high and since it is a mountain, there is no hindrance to the wind flow.

Solution Overview: We are solving the power generation challenge created by a mismatch between the speed of the wind and the necessary rotational speed required to produce power by the turbine’s generator. We have access to several used car parts, allowing us to salvage or modify different induction motors and gears to make the system work.

We have two approaches we are taking. One method is converting the induction motor to a generator by removing the need of an initial battery input and using the magnetic field created by the magnets. The other method is to rewire the stator so the motor can spin at the necessary rpm.

Subsystems: Our system components are split into two categories: Mechanical and Electrical. All mechanical components came from a used Toyota car such as the wheel hub cap, serpentine belt, car body blade, wheel hub, torsion rod. These components help us covert wind energy into mechanical energy and are already built and ready. Meanwhile, the electrical components are available in the car such as the alternator (induction motor) and are designed by us such as the power electronics (AC/DC converters). We will use capacitors, diodes, relays, resistors and integrated circuits on our printed circuit boards to develop the power electronics. Our electrical components convert the mechanical energy in the turbine into electrical energy available to the residents.

Criterion for success: Our project will be successful when we can successfully convert the available wind energy from our meteorological data into electricity at a low cost from reusable parts available to the residents of El Durazno. In the future, their residents will prototype several versions of our turbine to pump water from the mountains.