Beat egg yolks and sugar together in a bowl with an electric mixer until fluffy and pale yellow, 1 to 3 minutes.
Heat milk in a large pot over medium heat, and whisk in the egg yolk mixture until smooth; boil for 1 minute.
Mix in the cracker crumbs; bring to boil, reduce heat to low, and simmer the mixture until thick, about 5 minutes.
Beat the egg whites in a glass or metal mixing bowl with an electric mixer until they hold stiff peaks; gently fold into the other mixture, and mix in the vanilla extract.
Set aside and let cool.