When using the grill, the top elements in your oven turn on, giving direct heat to your food. If the door is closed, the heat will have nowhere to escape and your food will end up cooking too fast. If you do have the grill on it's always recommended that you leave the oven door ajar, unless stated differently in your manufacturer's manual.
If you have the oven door closed and have grill mode on, the temperatures can be too hot and can cause overheating. Overheating can damage the rubber seals on your oven door.
Some ovens have a built-in cooling fan, which allows you to have the door closed. The cooling fan blows out air between the control panel and the door. Always leave this air free from obstructions.
If your oven doesn't have a cooling fan, smoke can start building up inside the oven which could cause a fire. Leave the door ajar for ventilation. If you feel that there's still a lot of smoke try turning your extractor fan on for extra airing.
Hopefully you can now feel more confident when using the grill in your oven.
Always refer to the manufacturer's manual for specific details.