Sensing Technology and Driver Monitoring Systems
The OSRAM Podcast: Episode #3 with Martin Wittmann
Welcome to The Photonstudio, the OSRAM podcast. My name is Dieter Schierer and I am an OSRAM employee in the digital communication department. I am looking forward to a new episode from the Fascination of Light section. Today the topic is sensor technology in cars also known as the ‘invisible' passenger. Imagine you order a taxi, drive to the desired destination, get out of the car and as soon as the car is around the corner you notice that you have forgotten your bag in the back seat. How annoying!
Or imagine that you have been sitting behind the wheel for hours and don't notice that you are not concentrating on the road because you are tired. But your car does notice, warns you or initiates the braking process as a precaution. In both cases, invisible light plays a key role. Sounds like science fiction, but it's not. But how exactly do the invisible passengers in the car work? And how do they make traffic safer today and in the future?
To find out, I invited my guest today to the Photonstudio. Martin Wittmann is Head of Marketing for Sensors at OSRAM's business unit Opto Semiconductors. In other words, for the area whose products use invisible light to detect our environment and thus enable fascinating new applications.
Dieter: Martin, welcome to the Photonstudio!
Martin: Thank you Dieter, I am very happy to be able to talk to you!
Dieter: Let's get right into the conversation. You are with us in the OSRAM Opto Semiconductors business unit, which is based in Regensburg. There you are the Marketing Manager for the Sensing division, right?
Martin: That's right!
Dieter: What does your professional life actually look like? What does a marketing manager do for this particular area?
Martin: Sensing is an important growth area for OSRAM Opto Semiconductors. And as Marketing Manager I am responsible for the successful positioning and marketing of our product portfolio together with a global team. I think a key to success here is close cooperation with our sales team and proximity to our customers. In other words, a clear customer focus. This enables us to identify market trends early on and stay one step ahead of our competitors.
Dieter: The word "sensing", what does it actually mean?
Martin: At OSRAM Opto Semiconductors we basically differentiate between visible and invisible light. We address both. Our products and solutions in the sensing sector also use both visible as well as invisible light, so-called infrared light, which is very important in the field of sensing. The choice of the wavelength of the light depends on the specific application, specifically what information our customers want to detect. An example: fitness bracelets or so-called smart watches use both visible and invisible light to determine our health values. Many other sensing applications, such as security surveillance, use infrared light at 850 or 940 nanometers. This wavelength range is the core wavelength range for sensing and we call it NIR (near infrared).
Dieter: You have given a very good example with these smart watches and fitness trackers, which many athletes use. I also have a watch like that because I jog and cycle a lot. And actually there is an emitter on the back, a light module which shines green. But how does it work? Can you explain why it absolutely needs such a green light and how can the light determine how high my pulse is right now?
Martin: Basically we have an emitter component and a detector component for health applications. The wavelength depends a little bit on which part of the body we measure. For the arm, green light is well established. For other measurement methods, for example at the ear, a different wavelength gives better results and higher accuracy.
What we at OSRAM supply is emitter technology and detector technology to measure the pulse as reliably as possible and also, for example, the oxygen saturation in the blood. We see a very large market demand especially in the field of Vital Sign Monitoring because people want to control their fitness. Especially in times of corona virus, the measurement of oxygen saturation in the blood is very important, and our technology provides added value in this area.
Dieter: Good transition to my next question. Now, in the corona virus period with restrictions or quarantine, many sports are just impossible. It is no longer possible to go to the gym and group sports are currently still prohibited. This means that many people are discovering for themselves running sports, for example, or getting on a bicycle. Then of course they want to see if they are getting fitter and what their improvement in training and performance is like. Smartwatches or fitness trackers with integrated sensors are of course particularly suitable for this. Does OSRAM sense an increased demand from customers for these modules, which we can install?
Martin: Yes, definitely. On the one hand, we can see that the fitness monitoring sector is in very high demand. On the other hand, we also see an increased demand for medical solutions that are even more sophisticated, for example to monitor the elderly population. We also see a great deal of demand in this area at the moment. There it is important to increase the measurement accuracy even further. These are requirements especially for our customers who develop the algorithms. But of course we are working very closely with them to develop such medical solutions, especially with Covid-19.
Dieter: We met in January, even before Corona, in Las Vegas at the CES 2020 (Consumer Electronics Show), where you gave me very exciting insights into the topic of sensing for smartphones. We were standing in front of such a big screen, where various applications were explained via touchscreen. For example, you hold your phone up to a tomato or an apple and the phone can tell you how fresh this piece of fruit is and how many calories it has. Meaning that with spectroscopic methods you can determine the freshness of food. In the automotive sector there are so-called DMS systems, i.e. Driver Monitoring Systems. Could you perhaps tell us a little bit more about what these mean and how they are represented here?
Martin: Driver Monitoring is a new safety and driver assistance application that we see in the car interior. The aim here is to monitor the driver's attention, for example to detect the onset of fatigue. In some luxury class models, this function is already available on the road today. The systems are based on infrared lighting, a camera with a so-called CMOS image sensor and image analysis software that ultimately analyzes the driver's video. OSRAM Opto Semiconductors is the leading manufacturer for infrared lighting. Infrared emitters, so-called iLEDs, can be used for this. These have been on the market for several years and we are one of the main suppliers. Recently we also see an innovative laser technology, the so-called VCSEL technology, which is now in demand and used by manufacturers.
Dieter: You have now mentioned quite a few terms and also briefly described the processes. But let’s imagine a situation: I'm at the wheel and these modules are built into the cockpit. A light emitter sends out an infrared light, then there are CMOS cameras, like a sensor in a digital camera, and then you need special software that uses artificial intelligence to detect whether the driver is just yawning or almost falling asleep and perhaps causing an accident. Or how can we imagine that?
Martin: It's a lot about algorithms. Ultimately, your pupils are monitored, your head tilt and your eyes are followed. The algorithm detects whether you are gradually getting tired and gives you a warning signal. In the worst case, the car could intervene if it detects that you are no longer concentrated. There is a lot of know-how in the algorithm and partly also artificial intelligence to carry out this driver monitoring via software. The homogeneous and efficient illumination of the area that the camera captures is very important. And this is one of our strengths, to create the basis for a reliable system.
Dieter: To understand only briefly, in this process we are a supplier of light emitter systems. This means that the camera sensors, the software and the algorithms then come from customers with whom we work together?
Dieter: In some luxury class car models, the technology is already built in. If this really does contribute greatly to the safety of the driver and of motor traffic in general, can we expect that in a few years it will perhaps become the normality and the obligation that all cars must have this system in order to be registered at all? Or will it always remain am add-on feature that you have to acquire with more money?
Martin: No, absolutely not. We can see that over the next few years this technology will move from the upper class to the middle class. On one hand, we are working on making the systems a little more cost-effective. That is of course an important factor, but otherwise we can also see that safety standards such as the NCAP standard in Europe are pushing these safety systems very strongly and are also creating an incentive. At the end of the day, the great aim of all of us is to minimize the number of road deaths and to reduce the number of accidents. And often the cause of accidents is attention related. One gets distracted, keyword mobile phone etc. And these safety systems really help you concentrate and keep your eyes on the road.
Dieter: Can you give me an example? How will the system warn a driver if you look at your mobile phone? Will the car give a warning sign or does the steering wheel vibrate, or how can you imagine that?
Martin: The thing is, these days there's an audible signal. Typically, there is a warning signal on the display, and there may also be a vibration, for example, in the steering wheel. Right up to the point of actual intervention, so that the on-board computer takes over and brakes the vehicle. Various different scenarios are possible, depending on how inattentive the driver is. But step one would be an acoustic signal or a signal on the display.
Dieter: Martin, you mentioned the word VCSEL. It stands for Vertical-Cavity Surface-Emitting Laser. Can you explain exactly what this is and what it does?
Martin: VCSELs are a key technology for the rapidly growing field of 3D sensing applications. We are currently seeing a real boom for this technology, especially through its use in smartphones for 3D face recognition, for example. But this 3D recognition with VCSEL will sooner or later also find its way into the automotive sector. One option, for example, is driver identification in cars, similar to what we see with mobile phones. The vehicle could then automatically set your desired climate, the right seating position or load your favorite playlist. The bottom line is an improvement in your driving experience and comfort.
Dieter: In other words, if I want to unlock my car without a key, I look at a certain spot and the car says “hello, Dieter!” and I get in. And on the other hand, you can probably also save profiles, as it is the case with the phone or smart home. Then, when someone else sits down, the face is scanned and another one, for example, loads another music playlist. So the car behaves differently because another driver is there or is that too futuristic?
Martin: No, this is right! It even changes up to the driving behavior, for example if you drive a bit faster than your girlfriend.
Dieter: Many cars now have voice commands, for example, turn down the music, turn up the volume, control the navigation and so on. Is it possible to use VCSEL there to control certain commands in the car by hand movement? For example, that a certain hand movement stands for turning down the volume?
Martin: This is a good example and an interesting topic. There are already simple gesture controls directly on the display. We see this area as interesting for the future, especially when we think about autonomous driving and autonomous concepts. At the moment, I can control a lot at the steering wheel, for example, the volume or jumping to the next station. But I think in the future, when you get away from the steering wheel and drive really autonomously, gesture control will be very interesting because I won't have to reach for the center console anymore. With 3D sensing I can detect gestures very precisely, for example a hand movement to switch to the next radio station or to regulate the air conditioning with a hand movement. We find this really interesting and these are the trends we hear from automotive OEMs. VCSEL technology is predestined to make the whole thing in the vehicle even more intuitive, because I need a very high measuring accuracy to be able to control it reliably and cleanly.
Dieter: But what would be the advantage compared to the normal voice command, which everybody already executes anyway? Why do you need it now?
Martin: I think the car manufacturers definitely want to offer a certain redundancy to the customers and not just rely on one path. We are already seeing this in various areas, for example in mobile phones, to allow two paths. We also see that the systems complement each other well. For example, the voice control system has weaknesses when I speak and the radio is on at the same time. This interferes with voice recognition, for example. And if I drive the whole thing with a camera system, we can really see whether the command comes from the driver or whether it's perhaps a word from the radio. So I can optimize between the systems again and make sure that the whole thing works reliably and that there are no problems with the steering. I think it is very important for the driver or the owner of the car that such functions work reliably. And that's where we see the redundancy here as an advantage. Voice control plus the topic of 3D sensing.
Dieter: This means that if something is said on the radio station about the debt brake and the system misunderstands it and unintentionally initiates a braking process, we tend to resort to gesture control, don't we?
Martin: Right, of course that shouldn't happen.
Dieter: Martin, I think if the listeners have listened to us and followed the whole conversation, they automatically ask themselves whether the light does not damage their eyes? How do we make sure that, if I am constantly illuminated with this light, it is not harmful to the retina and eyes?
Martin: First of all, let's talk about eye safety, because it is clearly our top priority. It is very important that we guarantee eye safety and also meet the corresponding standards together with our customers at the application level. The infrared light in these applications is emitted over a large area, i.e. in contrast to a point laser, the infrared light is spread out. This means that the radiation densities are significantly lower and generally not critical for the human eye. In addition, these infrared light sources are pulsed so that we can further reduce the radiation power. This means that they are not continuously on, but pulsed. We also work very closely with our customers to jointly address these application aspects and ensure that the systems are really safe and that there is no damage to the occupants in the car. There are a few interesting aspects to the wavelength issue. First, the wavelength depends on which detectors we use. Typically, the detectors are most sensitive around 800-850 nanometers, but at this wavelength around 800-850 nanometers we still see a slight glow of these infrared LEDs and this is usually undesirable. You have to imagine that if you are sitting in front of the steering wheel and you have a slightly glowing source right in front of you, it is rather undesirable, and of course the car manufacturers don't want that. That's why we are going to the 940 nanometers, which are really not visible to the human eye. So aspects like that also play a part. As you can imagine, automotive interior is sacred and it has to look tiptop, even if maybe the camera technology would be a bit more sensitive at shorter wavelengths. But we are relatively flexible, depending on whether it interferes or not, we offer two different wavelengths.
Dieter: One last question from me on this subject. Is it also possible to fantasize further and say, for example, if I get out of the car in a taxi and forget my bag, that sensing in the car can be used to scan for objects and then recognize that there is another object that belongs to the passenger? Would such a direction also be conceivable, and would that be of interest for car manufacturers?
Martin: Absolutely, this is the third area of interest. We differentiate between driver monitoring, the topic of gesture recognition, and the number three is so-called in-cabin monitoring, where we can monitor the entire interior of the vehicle. And here it is quite possible to detect even small objects on the back seat. This is possible with the camera system and the accuracy we can implement. We are talking about a few millimeters and it would be absolutely possible, if you forgot your wallet, that something like that can be detected or a quick check can be made at the end of your taxi ride for example.
Dieter: Are there already first tests? Without giving too much away can say where we stand?
Martin: There are first prototypes on the road, I can only say that much.
Dieter: I am curious to see in how many years I will not have to worry about my ID or my wallet after the taxi ride anymore. My very last question has become almost standard. You are a graduate electrical engineer and work for us as a marketing manager. This means that you have to do and understand a lot of technical stuff, but on the other hand you have to convince the customer with a great presentation that our products are great and not those of the competition. What do you enjoy most about your work? Is it the technical aspect? Is it talking to the customers or something of both?
Martin: Good question, fortunately many things. What I enjoy most is working with our key customers, working out new, innovative solutions together and opening up new markets. Many of the applications are new, as you just tried to find out a bit. And I think that the photonics business in general, but also the sensing business in particular, offers enormous potential for us. And I see that with our technologies we are enabling end customers to have more comfort, more security and a better quality of life. And I'm really enjoying this close cooperation with the customers in our global team. It's great every day.
Dieter: Martin, that fits in perfectly with the OSRAM mission. Our mission is to use light applications to make people's lives better and, as you say, safer, simpler and easier. Martin, thank you very much for your time today, it was an exciting conversation. Although I prepared myself well, many things were new today. Especially concerning the wavelength, I felt a bit like in physics class in high school and then a bit of science fiction. I thank you very much for your time and I'm looking forward to seeing prototypes with our technology in usage, which will give the drivers the best experience.
Martin: Thank you very much for the opportunity and stay healthy!
Dieter: In this episode from Photonstudio, Martin explained how infrared light makes driving safer and more enjoyable. You can listen to this and all other episodes of our podcast in German on iTunes, Spotify, Soundcloud, and Google Podcast. If you want to know more about modern driver assistance systems and innovative solutions that are possible with infrared light, I recommend the online version of our innovation magazine ON. At www.osram-group.com/innovation you will also find many other exciting articles from the world of photonics. Have fun and see you in the next episode of Photonstudio!