Ytukyg

it seems to me that it is possible the US could pass a law that is
literally impossible to comply with. What would happen if it did?

The court would need to ascertain (1) the legislative intent, (2) whether it or the statutory language is unconstitutional, and/or (3) whether it is enforceable.

Apropos of unenforceable laws, see In re Initiative Petition No. 364, 930 P.2d 186, 201 (1996)

Unenforceable law is the very antonym of an initiative-authorized legal product. Proposing for adoption (through the initiative process)
a measure that is facially incapable of application as a state law is as much an oxymoron as "gentle cruelty" or "deft clumsiness."

(italics in original)

A scenario of the sort of "solving" the absurd equation 0x=50 ought to be held unconstitutional insofar as it would be "nothing more than a state-compelled false statement" (see Entertainment Software Ass'n v. Hatch, 443 F.Supp.2d 1065 (2006), which was affirmed) with obvious infringements of people's First Amendment rights. The only possibility of survival of that requirement would be if it were evident that legislative intent contemplated a special arithmetic where the zero-element differs from the role that number 0 has in traditional arithmetic.

Jury Nullification

While I'm sure this won't be a popular answer, it's worth noting that the founders of the United States solved this problem by requiring a jury trial. A jury has the freedom to not convict if someone breaks a stupid law, even if they know the person on trial did break the law. This is known as jury nullification, and merely bringing the idea up in a court is generally sufficient for you to be banned from the jury.

There may be a purpose to have laws which are impossible to follow.

(I'm neither a lawyer nor a politician, following points are what I like to call qualified hearsay - they come from qualified people I know personally but were given as a remark or during a chat over a cup of coffee and therefore are not easily substantiable with rigorous sources. You can treat them as a hypothetical ideas for your thought experiments.)

Everybody is implicitly guilty

Confident citizens and transparent law is the worst enemy of totalitarian regime. You learn to live with ingrained feeling that there surely is something you are guilty of. Merely being addressed by police makes you nervous and malleable; should you stand up against oppression, it is easy for the state apparat to detain or convict you of one or more default offenses.

A good example would be the law present in many, if not all, socialist bloc countries saying that knowing of a comrade having commited an offense or merely planing to and not reporting it to authorities is an offense in itself. Whether you did or did not know would be determined by the authorities.

Make your laws very strict with a hope thay they will be followed at least to a degree

Not laws in themselves, but standards (technical norms) regarding nuclear power stations in the former Soviet Union were strict to the point where they were technically impossible to follow given the state of the art. For example the standards for manufacture of high pressure pipes would state very low level of material impurities that when the actual manufactured material contained twice the level of impurities the pipe will still be very safe to operate. In a centrally planned economy with ever more ambitious production projections and declared zero need for contingency this was one of several ways how to create a bit of a wiggle room. (Source: I once worked for a nuclear power research institute supporting Soviet technology and was told this by an expert on stainless steel.)

So there you have a bit of an illustration what may happen if a law is intentionally impossible to follow. Since you labelled your question 'United States', I believe the follow-up question is why would anyone want to propose such a law.

Firstly, all laws passed by congress are done for public policy reasons. Therefore, for your hypothetical, assuming that Congress knew that it was impossible for some citizens to answer the question, there must be a public policy reason to make it impossible to comply. Perhaps they are trying to get rid of the mathematically illiterate? The thing is that impossible to follow laws have been passed and defended in court in the past. Below is a link to one such incident.

https://reason.com/blog/2018/06/29/californias-top-court-impossible-laws-no

There is nothing wrong with making impossible to follow laws. There are laws that get passed that make businesses go bankrupt because they cannot take the heat. The question is: are these impossible to follow laws constitutional? If they are then they stand in court. If they are not then they fall apart in court. I would argue that in the case of your hypothetical it would be unconstitutional. It would deprive the mathematically literate of rights granted to other citizens. It would also not be ADA (americans with disabilities act) compliant. Many disabled people are by nature of their disability mathematically illiterate. There are probably other reasons why it would be unconstitutional but nothing comes to mind.

EDIT: It has come to my attention that the equation is literally impossible to resolve. I suppose the article is less applicable as a result. However my point stands. Being impossible to follow changes nothing. All that matters is the intent behind the policy. In the hypo Congress intends NOT TO JUST IMPRRISON EVERYONE BUT INSTEAD TO CAUSE THE NATION TO COLLAPSE. I'm sure the courts would find that unconsitutional. On the other hand, suppose Congress were to pass a law saying "Peanuts are banned unless they were acquired from the core of the sun". That is impossible. Congress' intent would be to simply ban peanuts outright. Sounds pretty constitutional to me. Therefore it will stand in a court of law.

It gets ignored.

This is not a hypothetical; just do a search for "dumb laws" and you'll find hundreds of examples of laws that are either impossible or impractical to enforce. My personal favorite is the law prohibiting dogs from chasing cats (because we all know how law-abiding dogs are!). Of course, I don't know how many of these are real laws and how many were just made up for amusement; it takes a bit of effort to track them down. Still, a surprising number of them are real.

There are also a lot of old laws implemented at a time when it made sense, that are now meaningless, or that were nonsense to begin with. The most famous example of this, of course, is the english law making witchcraft illegal, which managed to stick around (in one form or another) right up until 1982. As far as I am aware, it is still illegal for MPs to wear armor in parliament, while the Metropolitan Police Act 1839 makes it illegal to play annoying games, or to shake a rug in the street. Yes, you read that right; it is illegal to be annoying.

Frankly, at this point the laws have become so byzantine and ridiculous, that I guarantee you break at least one every week.

A couple of years ago, a new law was passed that outlawed scented candles. Needless to say, nobody is bothering with it. Laws only matter if they are actually enforced, and the police are far too busy tackling real crime to waste their time on every crazy idea dreamed up by politicians with too much time on their hands.

Sorry my examples are all from the UK, but I think my point stands in general.

One common consequence of laws that are impossible, or very difficult to comply with, is increased opportunities for graft and bribery. Just give the local LEO or court official a gift and he will document that you have complied.

Sometimes such laws are enacted for this very purpose. Fortunately that is more common in other countries than in the US currently.

In the United States, the Eighth Amendment's prohibition against cruel and unusual punishment has been interpreted as prohibiting the government from criminalizing behavior that cannot be avoided.

See, most recently, Martin v. City of Boise. Here, the Ninth Circuit Court of Appeals held unconstitutional a law that prohibited people from sleeping in public. If you do not have a home and cannot obtain one, it is impossible for you not to violate this law.

Laws That Are 'Impossible' to Follow Can Still Be Constitutional, Says California Court

California passed a law a decade ago that demanded gun manufacturers implement microstamping technology that would imprint identifying information on bullets as they were shot from semi-automatic weapons. Gun manufacturers say the technology hasn't advanced enough to comply with the law. Smith & Wesson announced in 2014 that they would be pulling some guns from the market in California rather than complying with the law (a cynic might theorize that this is the law's actual intent).

California's Civil Code § 3531 says

The law never requires impossibilities.

So it is possible for legislatures to pass impossible laws. The court says they won't invalidate the law, but leaves room for special exemptions for punishment from those laws.

The court does not suggest that people can face punishment for being unable to comply with impossible laws. Instead, the court says, "impossibility can occasionally excuse noncompliance with a statute, but in such circumstances, the excusal constitutes an interpretation of the statute in accordance with the Legislature's intent, not an invalidation of the law." Essentially, it's not unconstitutional to pass impossible laws, but the courts can exempt people from the consequences of those laws without overturning the laws themselves.

The court acknowledges its role in making sure that people are not punished for being unable to comply with a law because it's impossible—that would be an unconstitutional violation of a person's rights. It just can't use that basis for invalidating the law itself.

So the court, depending on its bias, will leave an impossible law on the books, but require people who violate it to challenge the law every time in order to be excused from the punishment.

For your specific example, the government cannot compel speech. So providing a mathematical solution would be a form of compelled speech (as is writing computer code). So not only are you protected from punishment for what you say, you cannot be forced to say things the government wants you to. (There is also the 5th Amendment against compelled testimony against yourself.)

Compelled Speech

The compelled speech doctrine sets out the principle that the government cannot force an individual or group to support certain expression. Thus, the First Amendment not only limits the government from punishing a person for his speech, it also prevents the government from punishing a person for refusing to articulate, advocate, or adhere to the government’s approved messages.

In West Virginia State Board of Education v. Barnette (1943), the Court ruled that a state cannot force children to stand, salute the flag, and recite the Pledge of Allegiance.

Faced with this in a local law, I went to court, and didn't argue constitutionality, nor legislative intent. Rather I argued practical difficulty.

I will not elaborate more, because I do not understand law, in this instance, well enough to do so. But for the lower court, it worked and the state did not appeal the matter.

Addendum: Let me give an example which my research came up with when preparing my case.

Issue: When a vehicle is stopped to discharge a passenger, must the driver remain at the controls, while the passenger is being discharged? The zone permits stopping for loading and unloading [of passengers, cargo]. In this instance, a passenger is a 92 yo female who needs assistance entering and exiting the vehicle, but is capable of self-ambulating.

Rules: A loading zone exists to load and unload passengers, and a motor vehicle may not remain in the zone for any longer than is required to actively load or unload the motor vehicle. A stopped motor vehicle must have a driver at the controls, otherwise the vehicle shall be deemed parked. Also, a driver may not leave a motor vehicle engine running when the driver is not at the controls.

Analysis: It is normal practice for medical transport drivers to provide assistance to mobility limited passengers, and such assistance is essential to safety and efficiency at locations like train stations and airports when discharging or onboarding passengers. While the strict interpretation of stopping may not apply if the driver leaves the controls for purposes of assisting a passenger, it is the most efficious manner to assure that the rider's needs are being met, and that they safely board and unboard the motor vehicle. Furthermore, if mobility impaired riders were forced to use other loading locations, then they would normally be required to ambulate further, from a more distant location. [Or in the alternative, the majority of people would have to travel further to load and unload, to make a more desirable location available for mobility impaired passengers.] (Also included in the argument/analysis was much about ADA, state disability laws, equal access, etc.)

Conclusion: Where a mobility impaired passenger, who needs assistance boarding or unboarding a vehicle, the driver of that vehicle is most familiar with their needs and can efficiently assist the rider. This is commonly done by bus drivers, and provided for under the city transportation rules for bus drivers. Therefore, in the case of automobiles and other motor vehicles, there is practical difficulty, if the driver does not temporarily abandon the vehicle controls, and assist the passenger. To not do so, may require replanning of loading zones, inconveniencing the majority of passengers, and may violate the rights of mobility impaired passengers, in likely violation of ADA and state laws.

Result: County court ruled that assistance to passengers when stopping to load and unload passengers did not violate motor vehicle code regarding stopped vehicles.

Many US laws are already impossible to follow because, quite simply, they are impossible to know about. For example, a child born in France to US citizen parents is legally required* by US law to report all of his bank accounts to the US government. This despite the fact that he may have never left France in his entire life, never studied US law and does not have an American accountant in a 200 mile radius. The penalties for not doing so include massive fines and jail time.

There simply is no way the child could have known about these legal requirements, yet could and most likely would be held responsible for not complying - including by the French government who have agreed to prosecute French citizens and residents who do not comply with these US laws. (see also "Accidental Americans")

_{* assuming a net worth of over USD $10,000, which is essentially every adult}

This does happen. For example, the state of California has passed a law requiring certain firearms to have a feature that is not simply uneconomical, but actually impossible to manufacture with current technology. In effect, this is an outright ban on the affected types of firearms. Apparently the legislators believed that this trick would protect the law from Constitutional challenges, and so far they have been right. The law was upheld by state courts, and AFAIK, opponents have not challenged it in Federal courts, possibly due to the current fickle judicial environment. But this could change overnight depending on future appointments to the Supreme Court.