How can I use OOTB functionality to limit the escalation rule to only occur once?

Also if a workitem is still open, close it out after that attempt has been made?

There are two options/ways for this

1) If you need only a single escalation and then the workitem needs to stay at that level,
a) launch approval workitem for first approver, configure escalation workitemconfig as required.Here the escalation rule should return null to move the workflow forward.
b) launch the approval for next level. If you do not need escalation at this level do not configure the workitemconfig parameters here.This will make sure the workitem stays at this level till the approver approves the workitem.

2) If escalation is needed at both levels and if after second escalation you want to close the workitem do like this.

Declare a global variable called escalationLevel with the initialvalue of 0
Launch approval workitem with workitemconfig parameters.Within the escalationrule implement the following logic.
Context.getworkflow.getVariables.getInt("escalationLevel");

If escalationLevel value is 0
{
escalationLevel to 1 (Context.getworkflow.getVariables.getMap().put("escalationLevel","1")
rule should return next level approver
}
otherwise null.