Summary
We've had some researchers pass on reports from participants who say that they've received more than one announcement email for the same study/child, which should never happen. (These cases were reported by participants because the family were confused about whether they can/should participate again.)
We've also had reports from researchers that they're seeing the same user/child schedule/participate in their study multiple times. While most families probably understand that they'll only be compensated for a study once per eligible child, it's also true that, If the researcher does not add the study itself to the "must not have participated" list, then the family will not see the eligibility warning when they attempt to participate again with the same (otherwise eligible) child.
But another possible explanation for the duplicate participation issue could be that the family received multiple announcement emails for that study/child, and therefore figured that they must be allowed to participate again.
Possible causes and results of investigating
1. We've sent out multiple emails to the same study/user/child ❌
I've ruled this one out empirically. We have a messages table in the database with every announcement email ever sent, and there are no repeated messages for any study/user/child combination.
I also made sure that the message rows are actually unique to each message-send attempt (i.e. we're not re-sending any messages to the same study/user/child and just updating the existing row), and that is the case.
I also ruled out the possibility that we've had an issue with our announcement email task running more than it should, as simultaneously-running tasks could cause duplicate emails. But the task scheduling is not duplicated and the task's run count almost exactly matches the number of days since the first run (minus a little downtime), so the task is definitely only running once per day.
Here are some other possibilities for the above reports that don't involve multiple announcement emails but could cause the same sort of confusion.
2. User has multiple accounts
If the a single user has multiple CHS accounts, that user might get announcement emails about the same study sent to their different user (email) accounts. This would be especially confusing if (1) they have the same child(ren) added to both accounts, and/or (2) they've created multiple accounts using an email alias that sends email to the same inbox, e.g. the gmail +/.-in-username trick, as that would cause the emails sent to separate user accounts to arrive in the same inbox.
3. User has multiple children who are eligible at different times
When we send study announcement emails, we send a single email to the user that mentions all of their eligible children. However, eligibility changes over time (e.g. because a sibling ages into the age range), so a family might receive another email for the same study, but for a different child (because that child has just become eligible). This could be mistaken for multiple emails, and could be especially confusing if the parent is not paying attention to the child names in each email.
3. Child participates in a jsPsych study and then gets email invitation for that study
For jsPsych studies in particular, there is a bug in our study announcement logic - the user/child is not excluded from the list of potential email recipients if they have already participated in the study. This means that, if a user/child finds a jsPsych study on the study list and then participates (gets past the consent trial), the same user/child could still get an announcement email for that same study because the query did not exclude them. (But again, we never actually send a duplicate announcement email in this case - the confusion comes from the fact that the child has already done that study, and then they get an email about that study, which makes it seem to them like they're still eligible.)
4. The study is cloned and re-launched
In our study announcement logic, a "study" is defined/ID'd by its UUID, but study names (and any other info, like description, images, etc.) do not need to be unique. So if a researcher were to clone a study and re-launch it without changing much about the participant-facing aspects (name, description) then the family would still (correctly) receive one email per eligible child about each of these studies. If the study name was the same then that would appear to the family as an exact duplicate email, and if the study names were similar then it could easily be mistaken for a duplicate.
Other issues
There were some other issues that came up while I was looking into this that do not explain the user/researcher reports, but are still worth addressing:
- There's a tiny unclosed-loop in our process that could result in duplicate emails
When a new announcement email message is triggered, a new row is added to the messages table in the database with a NULL timestamp. Then the system sets up the message, triggers the message sending process, and then updates the timestamp for that row. When our task looks for potential study/user/child combinations, it excludes any message rows where the timestamp is NULL (it assumes that these cases failed). This means that, if a message row is created and the send is triggered, but then it crashes before the timestamp is set/saved, it will be treated by our system as "never sent" and the user .
The good news is that this scenario would have to leave behind a message row with a NULL timestamp, and I can see in the database that has almost never happened (one case from 2021).
- Preview responses count toward the participation exclusion
If a researcher uses a single account for experimenter and family purposes, and they preview a study with their real child profile, it will suppress the announcement email for that child. This is probably not what we want, since a researcher previewing a study is not likely to have their child with them, so that child should still be eligible. In other words, this is an issue that works in the opposite direction to the bug we're after (too few email targets rather than too many).
On the other hand, researchers probably don't want a bunch of study announcement emails for their fake/test children. This is a good reason (among others) to allow researchers to mark their associated child account(s) as fake/test, so that we can (a) stop treating preview responses as "real" study participation in our announcement email query, and (b) exclude any fake/test children from the email query.
Summary
We've had some researchers pass on reports from participants who say that they've received more than one announcement email for the same study/child, which should never happen. (These cases were reported by participants because the family were confused about whether they can/should participate again.)
We've also had reports from researchers that they're seeing the same user/child schedule/participate in their study multiple times. While most families probably understand that they'll only be compensated for a study once per eligible child, it's also true that, If the researcher does not add the study itself to the "must not have participated" list, then the family will not see the eligibility warning when they attempt to participate again with the same (otherwise eligible) child.
But another possible explanation for the duplicate participation issue could be that the family received multiple announcement emails for that study/child, and therefore figured that they must be allowed to participate again.
Possible causes and results of investigating
1. We've sent out multiple emails to the same study/user/child ❌
I've ruled this one out empirically. We have a messages table in the database with every announcement email ever sent, and there are no repeated messages for any study/user/child combination.
I also made sure that the message rows are actually unique to each message-send attempt (i.e. we're not re-sending any messages to the same study/user/child and just updating the existing row), and that is the case.
I also ruled out the possibility that we've had an issue with our announcement email task running more than it should, as simultaneously-running tasks could cause duplicate emails. But the task scheduling is not duplicated and the task's run count almost exactly matches the number of days since the first run (minus a little downtime), so the task is definitely only running once per day.
Here are some other possibilities for the above reports that don't involve multiple announcement emails but could cause the same sort of confusion.
2. User has multiple accounts
If the a single user has multiple CHS accounts, that user might get announcement emails about the same study sent to their different user (email) accounts. This would be especially confusing if (1) they have the same child(ren) added to both accounts, and/or (2) they've created multiple accounts using an email alias that sends email to the same inbox, e.g. the gmail +/.-in-username trick, as that would cause the emails sent to separate user accounts to arrive in the same inbox.
3. User has multiple children who are eligible at different times
When we send study announcement emails, we send a single email to the user that mentions all of their eligible children. However, eligibility changes over time (e.g. because a sibling ages into the age range), so a family might receive another email for the same study, but for a different child (because that child has just become eligible). This could be mistaken for multiple emails, and could be especially confusing if the parent is not paying attention to the child names in each email.
3. Child participates in a jsPsych study and then gets email invitation for that study
For jsPsych studies in particular, there is a bug in our study announcement logic - the user/child is not excluded from the list of potential email recipients if they have already participated in the study. This means that, if a user/child finds a jsPsych study on the study list and then participates (gets past the consent trial), the same user/child could still get an announcement email for that same study because the query did not exclude them. (But again, we never actually send a duplicate announcement email in this case - the confusion comes from the fact that the child has already done that study, and then they get an email about that study, which makes it seem to them like they're still eligible.)
4. The study is cloned and re-launched
In our study announcement logic, a "study" is defined/ID'd by its UUID, but study names (and any other info, like description, images, etc.) do not need to be unique. So if a researcher were to clone a study and re-launch it without changing much about the participant-facing aspects (name, description) then the family would still (correctly) receive one email per eligible child about each of these studies. If the study name was the same then that would appear to the family as an exact duplicate email, and if the study names were similar then it could easily be mistaken for a duplicate.
Other issues
There were some other issues that came up while I was looking into this that do not explain the user/researcher reports, but are still worth addressing:
When a new announcement email message is triggered, a new row is added to the messages table in the database with a NULL timestamp. Then the system sets up the message, triggers the message sending process, and then updates the timestamp for that row. When our task looks for potential study/user/child combinations, it excludes any message rows where the timestamp is NULL (it assumes that these cases failed). This means that, if a message row is created and the send is triggered, but then it crashes before the timestamp is set/saved, it will be treated by our system as "never sent" and the user .
The good news is that this scenario would have to leave behind a message row with a NULL timestamp, and I can see in the database that has almost never happened (one case from 2021).
If a researcher uses a single account for experimenter and family purposes, and they preview a study with their real child profile, it will suppress the announcement email for that child. This is probably not what we want, since a researcher previewing a study is not likely to have their child with them, so that child should still be eligible. In other words, this is an issue that works in the opposite direction to the bug we're after (too few email targets rather than too many).
On the other hand, researchers probably don't want a bunch of study announcement emails for their fake/test children. This is a good reason (among others) to allow researchers to mark their associated child account(s) as fake/test, so that we can (a) stop treating preview responses as "real" study participation in our announcement email query, and (b) exclude any fake/test children from the email query.