Technology has solved many of challenges, first world and not, but it has often left a large class of users behind. These are our elderly relatives, many of whom are the current the matriarchs and patriarchs of our families. They are the people who keep our history and know our legends. It is hard for them to easily capture their stories or for their to easily help them do so. Especially in cases where a family has undergone a schism or has had a tragic event resulting in lots of loss (World War II, etc.) we are on constantly seeing history lost to time.
How can technology help here? Especially with our most technologically challenged populations? If we think about how their challenges (vision, hand-eye coordination, inability to type) or fears and illiteracy manifest, we soon realize we have invented support for many of these. The biggest supports are the ability to zoom and the speech recognition and generation. They hypothetical grandparent in this scenario can talk to a digital assistant, perhaps one whose volume is turned up very LOUD and can see a zoomed in image on a high resolution screen. What we need is software that will combine these technologies in a way that allows our history keepers to do what they do best, tell stories. My vision is this. I want an application that can interview my grandmother. This may already exist, as many “novel ideas” do, but I haven’t seen it yet.
This application needs to be able to do the following:
Photo Scanning
Using the same ideas behind Google Photoscan, scan photos in place from framed pictures, albums, etc. Ideally, someone should be able to do this in a batch mode where everything is scanned at once, or serially where each picture is processed completely down this list after capture. Extend the existing applications to be album aware for bonus points. This means allowing the covers and non-photo pages of albums to be easily captured and stored in a related sequence. This minimizes the amount of meta-data required to be recorded at the time of the scan.
Additionally, capture the full page of all albums as often times photos only make sense in context. Then use something akin to face recognition technology to detect pictures on a full album page scan and then prompt for detailed scanning of each picture. This should automatically build an interface where you can virtually flip the pages of the album and then click each photo for details.
Interviews
Using voice recognition and voice generation technology walk through an interview. The interview should probably go something like this.
- What’s going on in this photo??
- What does seeing this make you think of??
- Where was this taken?
- When was this taken?
- Who’s in this photo?
This last question, especially should be combined with face recognition. Detect the faces and allow them to be identified one by one. As work progresses, shift the question to be one of “Is this Aunt So-and-so?” to avoid interview fatigue.
Challenges
Technology isn’t perfect, therefore the app needs to work around reality and be very forgiving. This means, for example, if a scan is discovered to be flawed during the interview, it should be able to be flagged for fixing later, not require external tracking of this fact or worse, a pause in the process. A lot of the target population has either limited patience or limited stamina. Do not interrupt them for any avoidable reason.
Voice recognition is definitely not perfect and I would not be surprised to learn it has a higher failure rate for elderly speakers. They probably have word choice, cadence, and accents that are not part of the general testing. Therefore record everything! Storage is cheap. If Grandpa’s voice is hard to understand, save his audio for either later reprocessing when the technology gets better or for his kids to manually work out (or correct the transcription) at a later time. Also record everything because hearing Grandpa tell the story is much more compelling than listening to Alexa/Siri/Cortana/Google tell it.
Bonus Points
-
Allow multiple people to be interviewed about the same photos. This way each generation and person adds their voice and memories. In many family’s there is no single truth.
-
Build a documentary. Your app can store bonus points by weaving together a story. The application now has massive amounts of who, what, where, when metadata and transcribed stories. I suspect with text generation and summarization technology a pretty good narrative could be written. Tweak the algorithms to retain key quotes and you could have a Ken Burns style documentary scripted and narrated by a generated voice with extensive direct quotes from the parties pictured.
Most of us have the goal of capturing our family’s history and stories, especially as our relatives age, however nothing has really made that easy. Technology should enable this goal in a solid way without requiring multiple gadgets and lots of “bookkeeping” off to the side.
