Simon MerrettOnce I had decided I wanted to synchronise my microcontroller with daylight levels as a proxy for the 24 hour clock, I needed to find a technique to get from sensor readings to a time of day.
Initially, this may seem like a problem of frequency measurement - we just need to measure how many microcontroller "periods", such as wake ups / timeouts / timer interrupts occur in a single cycle of daylight. But it is not quite that easy and I'll come back to this.
Pattern matching for basic microcontrollers
Before I get back to the main story, I want to cover some of the interesting "dead ends" I found while doing my initial searches for techniques to synchronise my data, which I anticipated would have clear periodic trends, using a microcontroller. If you are doing something similar or slightly different, this may be of interest.
Firstly, I looked for frequency detection/measurement on basic microcontrollers (by which I mean something like the ATMEGA328P would be a typical example). Loads of audio applications want to do something like this for instrument tuning or spectrum analyser and cool visual effects to accompany the music. Here are the sort of progressions you may find useful:
- Some kind of "basic" (not to take anything away from the achievement but just in terms of algorithmic complexity) peak detection, zero-crossing or non-zero-crossing technique, as masterfully demonstrated by Amanda Ghassaei in her instructible on the topic. https://www.instructables.com/Arduino-Frequency-Detection/
- Fast Fourier Transforms, to identify the "strongest" frequency component in a compound waveform (yes, I know, the are pretty much all compound!), and I like the blog post by Arik Yavilevich on his exploration and success in this area. https://blog.yavilevich.com/2016/08/arduino-sound-level-meter-and-spectrum-analyzer/
- Finally, and most promisingly, we have autocorrelation. This technique doesn't care about how many zeros you do or don't cross, what kinds of spectral composition you bring to the data but it only cares about repetition - that sounds good, right? Akellyirl has a great instructible with links and theory and library content on this. I highly recommend https://www.instructables.com/Reliable-Frequency-Detection-Using-DSP-Techniques/
But none of those seemed quite right when I dug deeper into dayTime.
The first issue is that we are not just measuring frequency of the daily light cycle. We want to synchronise, or lock-in to the phase of the cycle too. So this is one task or feature we need. The frequency detection didn't do this out of the box and the FFT did not do this at all. Autocorrelation and frequency detection might have been able to to phase lock with some more work.
The second challenge is that our readings are not going to produce nice regular patterns. This is obvious from the fact that we get dull and bright periods within each day (noise) and that the overall proportion of daylight in a 24 hour period varies from winter to summer in most parts of the world (seasonal). It is akin to a varying pulse width modulation duty, where the duty represents the hours of daylight and the fixed frequency represents our 24 hours:
Source: By Eighthave, modified by Teslaton - Modification of Image:Pwm duty cycle.gif, Public Domain, https://commons.wikimedia.org/w/index.php?curid=3674466
With wildly varying levels of light (large dynamic range) between winter/summer and within clear and stormy days, none of the options above looked like they would reliably identify daily periods and phase points.
Looking for something which would provide a phase locked loop, I found the wonderful #Software Phase Locked Loop project by @agp.cooper . Although I did not go through with a build based on this, I learned a great deal and would recommend you check out this project. Top tip: another helpful search term for you - "product detector" might be something you want for fast locking/correlation.
As I was researching all this, I learned that was I was really doing was "Time Series Analysis", which you may find a useful search term if you are doing something similar. I mean, if you just want an audio VU meter, you might not think to delve into Time Series Analysis. But it might be helpful.
Time series analysis has varied applications in data science so some of the web content can feel a little mathematical (because it is!) but some sites boil the main concepts down really well. I like https://www.aptech.com/blog/introduction-to-the-fundamentals-of-time-series-data-and-analysis/ because they give you a simple graph to explain each concept or term in a clear way.
For #dayTime , we are expecting noise / randomness in the brightness on the sensor at any particular time of reading and a degree of measurement noise within the sensor electronics. We are also expecting daily cycling of sensor values (seasonal) and annual changes in daylight hours (also seasonal but on a different timescale). We are not expecting a trend - ie we are hoping that days remain 24 hours long! But I suppose we should be open to the possibility of the microcontroller oscillator aging to the extent that it counts fewer or more "microcontroller periods" per 24 hour period.
But I'm getting ahead of myself - what data was I using to evaluate the techniques I considered above?
Discussions
Become a Hackaday.io Member
Create an account to leave a comment. Already have an account? Log In.
Hi Simon,
Cloud cover will of course affect the determination of dawn/dusk. This can only be reduced by multiple dawn/dusk (daily) estimates.
However, the period of interest is between sunset/sunrise (-0.833 degrees) and twilight (-6 degrees). This is a 21 minute window. Regardless of cloud cover you will detect sunset/sunrise etc. within a few minutes.
Can we do better?
There is an inflection point in the light level (which I am calling dawn/dusk as these two terms are not explicitly defined) that is between -2 degrees and -4 degrees. This is an 8 minute window. The inflection point shows up quite distinctly (more so than sunrise/sunset) in real data (using a smoothed second derivative):
https://hackaday.io/project/47132/gallery#65f75037543d67beee30669edc5b36ed
So cloud cover will delay dawn a little and advance dusk a little. How much is an open question.
At the moment I am using a 31 element digital filter that results in a 15 minute delay in processing. The size of the filter can be increased to separate the 24 hour cycle from the short-term(?) noise from clown cover.
Can we do better?
If your happy to point your sensor directly at the sunset/sunrise/noon (note it is a moving target!) then you certainly can do better. A sextant is such an instrument.
Regards Alan
Are you sure? yes | no
Hi Simon,
I had a read through your posts and I can offer some of mine real data (minute by minute measurements) but I did not save most of data I collected. The lDR is a great sensor for daylight as it is logarithmic and usable well into twilight.
Below is a link to some data I collected (where I tried to fit a LUX model):
https://hackaday.io/project/47132/gallery#e893e0bfabae6726eeb83b7fbf9f09f5
I have abandoned this approach however.
This one shows the second derivative peaks (miss-named as LDR' (green)) that I use to mark dawn and dusk:
https://hackaday.io/project/47132/gallery#d334dffbb3caddc482cf0440c77b3517
These peaks sit between sunset/sunrise and twilight and are very consistent even with noisy data. This is the approach I use and I would suggest.
But I can generate LUX levels for anywhere on earth at anytime. If that helps. The code is in C and XLS based.
I am pretty sure you cannot use the existing Arduino clock/timer directly but I have some code that piggy packs the same interrupt that models a real clock properly. The Arduino clock is off +/-20 seconds per day (at least mine was) so it needs a correction factor fro your device. The standard crystal oscillator is just not that accurate on the Arduino. So I can see the benefit of following daylight cycles. But you still need latitude/longitude and the date/year!
I suppose I should mention that the code is pretty complex if you have had no exposure to The Astronomical Almanac.
So in the end, the problem you want to solve is not simple but it can be done.
Regards Alan
Are you sure? yes | no
Thank you for reading! You are "spoiling" the ending! I have some LDR data and will be coming on to it in a log soon. I'm just trying to log in chronological order and have only caught up to the beginning of the project.
I will be sharing my saved LDR data, although I need to order it/sort it first. And it is as you say - nice for these purposes - but that is jumping ahead of me publishing some of the strange processes I went through to tame the W/m2 solar radiation data into day sync for the microcontroller.
Are you sure? yes | no
Hi Simon,
I was thinking of restarting my sunrise and sunset geo-location project ( https://hackaday.io/project/47132-daylight-geolocator-remix ) just a days ago.
It just needs a solar power and battery solution for the "box". I have worked out why the storage chip was not working.
---
I have looked at a similar problem of auto-calibrating geo-location using a real-time clock and a LDR.
In my case I found sunrise and sunset to have a very sharp response and quite feasible for accurate (less than 2 km or 4 seconds) geo-location calculations. The problem was defining sunrise and sunset mathematically (i.e. a negative altitude, between -2 and -3 degrees).
What I found was that over a series of readings that the result would drift in one direction if the altitude assumption was wrong. Therefore the drift can be used to auto-correct the altitude assumption.
The effective altitude assumption will vary for your site. A valley would be different to a flat area. Currently my code corrects for equal sunrise and sunset altitudes. I suspect a much better result could be had for differing sunrise and sunset altitudes.
---
Obviously, if you have the geo-location then the calibrated sunrise and sunset altitudes can be calculated based on the observations. Then the time estimate would likely be good to about a second.
Regards Alan
Are you sure? yes | no
Hi Alan, sorry for not replying this morning and thank you for your comment. I was trying to get another log or two under my belt before the dataloggin' contest closes today.
I will dig into your geolocation project - I'm sure there will be gems there too. Part of this project's aim is to get everything I have done out into the open and then we can pick over it and pull out the bits that work from a range of sources.
I have not read your project yet so forgive me if this is answered there but how might the time estimate you mention in your comment (based on light readings and a known geolocation) be affected by weather (cloudy morning followed by clear sunset, perhaps causing a phase offset in the rise/fall of the curve)?
Are you sure? yes | no