Elliptic curve cryptography sits under TLS key exchange, digital signatures, and secure messaging. Its safety rests on a quiet assumption: every point your code does math with actually lies on the curve you chose. An invalid curve attack breaks that assumption. If your implementation accepts a point from the other side and never checks that it belongs to the curve, an attacker can hand you a point that lives on a different, weaker curve. Over many exchanges, the math leaks your private key one small piece at a time.
Why a point has to be on the curve
A short Weierstrass curve is the set of points (x, y) that satisfy y^2 = x^3 + a*x + b mod p, plus a point at infinity. Your private key is a secret scalar d. In a key exchange you receive the peer’s public point P and compute a shared secret as the scalar multiple d*P, then run it through a key derivation function.
The catch is in the arithmetic. The formulas for point addition and doubling on a short Weierstrass curve use the coefficient a and the prime p, but they never use b. The curve constant b is only pinned down by the points you start with. So if you start with a point that does not satisfy your equation, the ladder still runs. It just computes the scalar multiple in the group of some other curve.
The math that does not notice
Say your curve uses constant b, but the attacker sends a point (x, y) that instead satisfies y^2 = x^3 + a*x + b' for some different b'. Your scalar multiplication routine reads x, y, a, and p, does exactly what it always does, and returns d*P computed on the attacker’s curve. Nothing errors out. The numbers look fine. You derived a key from a secret that now lives somewhere you did not intend.
Picking a curve full of small subgroups
Here is where it turns into key recovery. A well chosen curve has a large prime group order, so scalars are hard to guess. The attacker instead looks for a nearby curve whose order has small factors. On that curve they find a point P of small order r, say order 5. Because P has order 5, the result d*P can only be one of 5 possible points, and which one depends solely on d mod 5.
The attacker now learns d mod 5 by seeing which shared secret you ended up with. They do not see the secret directly, but they see something derived from it: an authentication tag that verifies, a handshake that completes, or an error that does not. Each completed exchange confirms one guess. Five tries at most pin down d mod 5.
Skip the on curve check and each handshake stops being a secret exchange. It becomes a yes or no question about your private key, and the attacker gets to ask as many as you will answer.
Then they repeat with points of order 7, 11, 13, and so on, collecting d mod r for many small coprime values of r. Once the product of those r values exceeds the size of d, the Chinese Remainder Theorem reconstructs the full private key. A few dozen chosen exchanges can be enough.
How it shows up in code
The bug is almost always a missing line. A vulnerable exchange reads the peer point straight off the wire and multiplies:
point = parse_peer_point(request) # x and y taken from the message shared = scalar_mult(our_private, point) # no on-curve check here key = kdf(shared) send_confirmation(key) # tells the attacker if the guess was right
The fix is the check that is not there: confirm the point satisfies the curve equation, reject the point at infinity, and confirm the coordinates are in range before you multiply by your secret. One validation call stops the whole family of attacks.
How to defend against an invalid curve attack
The defenses are concrete and they stack. None of them needs new cryptography.
- Validate every received point. Plug
(x, y)intoy^2 = x^3 + a*x + b mod pand reject anything that does not satisfy it. Also reject the point at infinity and coordinates outside[0, p). - Check the subgroup. Confirm the point has the correct order, or use a curve with cofactor 1 so a valid on curve point cannot sit in a small subgroup. Multiply by the cofactor and reject the identity when the cofactor is larger than 1.
- Prefer x only ladders. Montgomery curve designs like the one behind X25519 compute with the x coordinate alone and are built so that points map into the intended group, which removes most of the footgun. Use them where you can.
- Use a vetted library and its high level API. Hand rolled point parsing is where the check gets dropped. A reviewed library validates inputs for you, so let it.
- Validate even ephemeral keys. People skip checks on ephemeral ECDH because the key is short lived. The attack works per exchange, so a throwaway key still leaks bits. Validate every time.
This is a close cousin of the small subgroup confinement attack, where a valid point in a small subgroup leaks key bits the same way, and it rhymes with ECDSA nonce reuse, another case where sound elliptic curve math is undone by one missing usage rule. For more of these, see our hub on cryptographic oracle attacks, and the vulnerability basics category for the groundwork.
The assumption that breaks
An invalid curve attack is not a flaw in elliptic curve cryptography. The curve is strong and the scalar multiplication is correct. The failure is an input that was trusted without being checked, a point that was never confirmed to be on the curve before a secret touched it. That kind of gap hides in code review because the scalar_mult call reads as correct on its own line. Finding it means asking what the code assumes about its inputs and then testing whether an attacker can make that assumption false. That is exactly the kind of reasoning an autonomous researcher that tests assumptions is built to do. Read more on our about page.
Frequently asked questions
What is an invalid curve attack?
It is an attack on an elliptic curve implementation that accepts a point without checking it lies on the chosen curve. The attacker sends a point that belongs to a different, weaker curve. Scalar multiplication still runs because the point addition formulas do not use the curve constant b, so the secret key gets multiplied into a group the attacker controls, leaking key bits over repeated exchanges.
How does the attack recover the whole private key?
The attacker sends points of small order, so the result of the scalar multiplication depends only on the private key modulo that small order. Each completed exchange confirms the key value modulo a small number. By collecting the key modulo many small coprime values and combining them with the Chinese Remainder Theorem, the attacker reconstructs the full private key from a few dozen chosen exchanges.
How do I prevent an invalid curve attack?
Validate every received point before you multiply it by a secret. Confirm it satisfies the curve equation, reject the point at infinity, and check the coordinates are in range. Verify the point is in the correct subgroup or use a curve with cofactor 1. Prefer x only ladder designs like X25519 and use a vetted library that validates inputs for you.
Does using an ephemeral key make my code safe?
No. The attack works one exchange at a time, so a short lived ephemeral key still leaks bits of whatever secret touches the untrusted point. People skip validation on ephemeral keys because they feel disposable, but you must validate every received point every time, ephemeral or not.
Put an autonomous researcher on your own systems
UnboundCompute is an autonomous security researcher that reasons about how an application fits together and proves the access control and injection bugs it finds. We are opening a small number of founding design partner seats: private early access pointed at a staging target you choose, and a say in what it looks for. If your team ships software worth pressure testing, apply to the design partner program.