Carbohydrates are cheap. I am on a low-carbohydrate diet and my fellow LC dieters often like to argue that there's a way to eat inexpensively on this diet, but I haven't really learned how to after 5 years. My food budget isn't astronomical, but I remember the days when a box of rice or a bag of potatoes would go a long way.
Strictly speaking, you are not on an LC diet, but because you're avoiding starches (and presumably most types of sugar?), we actually have a lot in common. So let me talk about what you can eat that hopefully won't bust your budget.
The foods you *can* eat are meat, cheese, eggs, vegetables, fruit, butter, olive oil, coconut oil, cream cheese, heavy whipping cream, plain yogurt (Greek or regular, although Greek is usually lower in carbohydrates), nuts, and seeds. If you are restricting sugar, stick to lower-carbohydrate fruits, like berries, avocados, tomatoes, and olives.
Now that you know the staples of your diet, you can more easily plan. I recommend buying large cuts of meat that you can roast or slow cook and then eat for days. For example, roast a chicken, have some chicken and vegetables for dinner, and then use the leftover chicken for chicken salad in a lettuce wrap the next day or for part of a stir fry for dinner the next night. A large pot of a meat and vegetable soup can be prepared cheaply and last for many meals (even if it's just a side dish the next night).
For breakfast, how about a frittata? I make a delicious one by sauteing spinach and mushrooms in butter in a non-stick pan with a metal handle that can be placed in the oven or under the broiler. When tender, I add my beaten egg and cream mixture. I use a spatula to move the liquid egg on top underneath so it'll cook, and when the bottom is setting up, I take it off the heat and add cheese. I like to use goat cheese, but it's quite expensive, so American, cheddar or whatever cheese you can buy inexpensively works. Place under the broiler for 30-45 seconds so the cheese with melt and the top will cook.
A bag of spinach cost me $1.50 and a tub of mushrooms cost me $1.99. The eggs cost me $2. The cheese cost me $4. I only use 1/4 the bag of spinach, 1/4 the mushrooms, 3 eggs out of a dozen, and an ounce of cheese, so I could make this exact breakfast many more times without needing to buy anything else.
Casseroles are also your friend. You don't need a starch in them. In fact, I'll link you to some of my cheap favorites. I'm also going to link you to something called Mr. Peanut Bread and a spinach-crust pizza. Seriously, look at these.